Sensor fusion to validate sound-producing behaviors

ABSTRACT

A method to measure sound-producing behaviors of a subject with a power- and bandwidth-limited electronic device that includes a processor includes measuring, by a microphone communicatively coupled to the processor, sound in a vicinity of the subject to generate an audio data signal that represents the sound. The method also includes measuring, by at least one second sensor communicatively coupled to the processor, at least one parameter other than sound to generate at least a second data signal that represents the at least one parameter other than sound. The method also includes detecting one or more sound-producing behaviors of the subject based on: both the audio data signal and the second data signal; or information derived from both the audio data signal and the second data signal.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/118,242, filed on Aug. 30, 2018, now U.S. Pat. No. 11,172,909, issuedon Nov. 16, 2021; the disclosure of which is incorporated herein byreference in their entireties.

FIELD

Some embodiments described herein generally relate to sensor fusion tovalidate sound-producing behaviors.

BACKGROUND

Unless otherwise indicated herein, the materials described herein arenot prior art to the claims in the present application and are notadmitted to be prior art by inclusion in this section.

Sound-related behaviors such as sneezing, coughing, vomiting, shouting(e.g., tied to mood or rage) may be useful to measure in health-relatedresearch. For example, measuring sneezing, coughing, vomiting, shoutingmay be useful in researching the intended effects and/or side effects ofa given medication. Such behaviors have been self-reported in the past,but self-reporting may be cumbersome to subjects, may be inefficient,and/or may be inaccurate.

Some methods have been proposed to automatically measure, e.g., sneezingand/or coughing. Such methods often have to be implemented in noise-freeor at least low-noise environments such as in a clinical setting. Suchconstraints may make such methods expensive and/or difficult to deployon a large scale.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is only provided toillustrate one exemplary technology area where some embodimentsdescribed herein may be practiced.

BRIEF SUMMARY OF SOME EXAMPLE EMBODIMENTS

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential characteristics of the claimed subject matter, nor is itintended to be used as an aid in determining the scope of the claimedsubject matter.

Some example embodiments described herein generally relate to sensorfusion to validate sound-producing behaviors of a subject.

In an example embodiment, a method to measure sound-producing behaviorsof a subject with a power- and bandwidth-limited electronic device thatincludes a processor includes measuring, by a microphone communicativelycoupled to the processor, sound in a vicinity of the subject to generatean audio data signal that represents the sound. The method also includesmeasuring, by at least one second sensor communicatively coupled to theprocessor, at least one parameter other than sound to generate at leasta second data signal that represents the at least one parameter otherthan sound. The method also includes detecting one or moresound-producing behaviors of the subject based on: both the audio datasignal and the second data signal; or information derived from both theaudio data signal and the second data signal.

In another example embodiment, a power- and bandwidth-limited system tomeasure sound-producing behaviors of a subject, includes a processor, amicrophone, at least one second sensor, and a non-transitorycomputer-readable medium. The microphone is communicatively coupled tothe processor and is configured to generate an audio data signal thatrepresents sound in a vicinity of the subject. The at least one secondsensor is communicatively coupled to the processor and is configured togenerate at least a second data signal that represents at least oneparameter other than sound. The non-transitory computer-readable mediumis communicatively coupled to the processor and has computer-executableinstructions stored thereon that are executable by the processor toperform or control performance of operations. The operations includereceiving the audio data signal from the microphone. The operations alsoinclude receiving the second data signal from the at least one secondsensor. The operations also include detecting one or moresound-producing behaviors of the subject based on: both the audio datasignal and the second data signal; or information derived from both theaudio data signal and the second data signal.

Additional features and advantages of the disclosure will be set forthin the description which follows, and in part will be obvious from thedescription, or may be learned by the practice of the disclosure. Thefeatures and advantages of the disclosure may be realized and obtainedby means of the instruments and combinations particularly pointed out inthe appended claims. These and other features of the present disclosurewill become more fully apparent from the following description andappended claims, or may be learned by the practice of the disclosure asset forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

To further clarify the above and other advantages and features of thepresent disclosure, a more particular description of the disclosure willbe rendered by reference to specific embodiments thereof which areillustrated in the appended drawings. It is appreciated that thesedrawings depict only typical embodiments of the disclosure and aretherefore not to be considered limiting of its scope. The disclosurewill be described and explained with additional specificity and detailthrough the use of the accompanying drawings in which:

FIG. 1 illustrates an example environment in which some embodimentsdescribed herein can be implemented;

FIG. 2 is a block diagram of a wearable electronic device and a remoteserver of FIG. 1;

FIGS. 3A-3I include various examples of audio data signal and variousextracted features;

FIG. 4 includes a flow chart of an example method to measuresound-producing behaviors of a subject with a power- andbandwidth-limited electronic device that includes a processor;

FIG. 5 includes a flow chart of another example method to measuresound-producing behaviors of a subject,

all arranged in accordance with at least one embodiment describedherein.

DETAILED DESCRIPTION OF SOME EXAMPLE EMBODIMENTS

Some embodiments described herein generally relate to sensor fusion tovalidate and/or measure sound-producing behaviors of a subject. Suchsound-producing behaviors can include sneezing, coughing, vomiting,shouting, or other sound-producing behaviors. The embodiments describedherein may detect sound-producing behaviors in general and/or maycategorize each of the sound-producing behaviors, e.g., as a sneeze,cough, vomiting, wheezing, shortness of breath, chewing, swallowing,masturbation, sex, a shout, or other particular type of sound-producingbehavior.

An example implementation includes a wearable electronic device, such asa sticker with sensors, smart watch or other wrist-wearable electronicdevice, etc. that leverages sensor fusion to collect sensory data fromdisparate sources to validate or detect behaviors or biometricresponses, such as sound-producing behaviors, with more certainty thanwould be possible when the sources are considered individually. Inparticular, the wearable electronic device may include multiple sensorsthat, at least in some circumstances, may be individually incapable ofdetecting sound-producing behaviors, at least with sufficient certainty.In aggregate, however, signals from the multiple sensors may be used todetect sound-producing behaviors, or to at least reduce the uncertaintyassociated with detection based on only one of the signals from only oneof the sensors. While described as being implemented in a wearableelectronic device, in other embodiments the various sensors may bedistributed across multiple discrete devices that provide their sensordata to the wearable electronic device.

The various sensors include at least a microphone. The various sensorsadditionally include at least a second sensor configured to measure aparameter other than sound. For instance, the various sensors mayadditionally include at least one of an accelerometer, a gyro sensor, aGlobal Positioning System (GPS) sensor, an oxygen saturation sensor(SO₂) such as a pulse oximeter, a thermometer, a photoplethysmograph(PPG) sensor, an electrocardiograph (ECG) sensor, and/or anelectrodermal activity (EDA) sensor. The wearable electronic device mayinclude the microphone and as few as one, and up to all of the othersensors and/or other or different sensors.

The microphone may be used to record sound. The recorded sound may beprocessed to extract features indicative of discrete sound-producingbehaviors, which extracted features may be used to detectsound-producing behaviors in general and/or specific types ofsound-producing behaviors. While the term microphone is used, it will beappreciated that term includes any type of acoustic sensor that may beconfigured to detect sound waves and convert them into a readable signalsuch as an electronic signal. For example, a piezoelectric transducer, acondenser microphone, a moving-coil microphone, a fiber opticmicrophone, a MicroElectrical-Mechanical System (MEMS) microphone, etc.or any other transducer may be used to implement the microphone.

The accelerometer may be used to measure acceleration of at least aportion of the subject, such as the subject's wrist upon which thewearable electronic device that includes the accelerometer may be worn.The recorded acceleration may be processed to extract features that maybe indicative (or not) of the sound-producing behaviors. For instance,when the subject sneezes, the subject's body (e.g., chest) upon which asticker is placed may move violently and/or the wrist upon which thewearable electronic device is worn may move quickly to the subject'smouth as the subject attempts to cover the subject's mouth during thesneeze. The violent movement of the subject's body (e.g., chest) and/orof the wrist as the subject's mouth is covered may be identified in theacceleration recorded by the accelerometer. Alternatively oradditionally, when the subject coughs, the subject's body (e.g., chest)and/or wrist may move less or not at all compared to when the subjectsneezes, which data may be used to help distinguish between a cough anda sneeze in the sound recorded by the microphone. Alternatively oradditionally, when the subject vomits, the subject may clutch theirbelly, a bowl, or make another movement that may be associated withand/or indicative of vomiting, which data may be extracted from theacceleration to confirm whether the subject has vomited. In someembodiments, the accelerometer may be used to measure the orientation ofthe body of the subject, such as whether they are walking, lying down,etc.

The gyro sensor may be used to measure angular velocity of at least aportion of the subject, such as the chest of subject upon which asticker may be placed or the subject's wrist upon which the wearableelectronic device that includes the gyro sensor may be worn. The angularvelocity may be used in an analogous manner as the acceleration toconfirm occurrence of sound-producing behaviors and/or to distinguishbetween different types of sound-producing behaviors based on measuredmovement (specifically angular velocity) of the subject's wrist or otherbody part to which the wearable electronic device is attached. E.g.,specific movements (like moving a hand to cover a mouth while sneezingor coughing) associated with specific sound-producing behaviors may beidentified in the measured angular velocity by extracting correspondingfeatures therefrom. In some embodiments, the gyro sensor may be used tomeasure the orientation of the body of the subject, such as whether theyare walking, lying down, etc.

The oxygen saturation sensor may be used to record blood oxygenation ofthe subject to generate a blood oxygenation level signal of the subject.Behaviors like shortness of breath or fits of coughing may correspond toa decrease in blood oxygen saturation levels.

Other sound-producing behaviors may also correspond to variations inblood oxygen saturation levels. In these and other embodiments, dataregarding the blood oxygen saturation levels may be used to extractfeatures from the associated data, such as decreases in blood oxygensaturation, etc.

The thermometer may be used to record temperatures associated with thesubject, including skin temperature and/or core body temperature. Insome embodiments, the temperatures may correspond to conditions such asfever or elevated skin temperature for infection or other conditionsthat may accompany one or more sound-producing behaviors. For example,the thermometer may indicate that when experiencing high amounts ofcoughing, the subject was also experiencing a low-grade fever.

The PPG sensor may be used to record blood flow of the subject.Behaviors like sneezing may be sufficiently violent in at least somesubjects that the jolt of a sneeze may manifest in the recorded bloodflow. Alternatively or additionally, heart rate may be extracted fromthe recorded blood flow. In these and other embodiments, vomiting and/ornausea may often be associated with elevated heart rate such that anelevated heart rate determined from the recorded blood flow may confirmoccurrence of vomiting.

The ECG sensor may be used to measure electrical activity of thesubject's heart to determine the subject's heart rate and/or otherparameters. Accordingly, ECG information recorded by the ECG sensor maybe used in an analogous manner as the recorded blood flow.

The EDA sensor may be used to measure EDA of the subject's skin. The EDAinformation may manifest extractable features for at least some of thedifferent types of sound-producing behaviors, such as at least vomiting.

Features may be extracted from the sound recorded by the microphone andfrom at least one other signal generated by at least one other sensor.When features from two or more of the sensors simultaneously indicate asound-producing behavior, a sound-producing behavior may be detectedwith more certainty compared to detection based on a single one of thesignals.

The embodiments described herein may be implemented using relativelycheap wearable electronic devices. Such wearable electronic devices mayhave a relatively constrained power supply (e.g., a relatively smallbattery) and/or a relatively constrained network connection (e.g., aBluetooth connection on the order of tens of megabits per second(Mbit/s)). In such a wearable electronic device, it may be too power-and/or processor-intensive to process the data locally on the wearableelectronic device using standard algorithms, and/or toobandwidth-intensive to upload the data to a remote server in the cloudfor processing. Additionally, it may be overly power-intensive tocontinuously record audio data using the microphone.

Accordingly, embodiments described herein may be implemented in a mannerin which the microphone may operate in a default “off” state in which itis not consuming power and not gathering data of audio around thesubject. As events of interest are detected by the other lesspower-intensive sensors, a signal may be sent to the microphone so itmay be turned on to capture audio data. For example, an accelerometersensor may detect a rapid acceleration of the chest (such as experiencedduring a sneeze, vomiting, etc.) and in response to the detectedacceleration, the microphone may be turned on for a three second window,after which the microphone may turn off again until another wakingsignal may be received.

Additionally, embodiments of the present disclosure may implementfeature extraction to extract relevant features from data signalscollected or received at the wearable electronic device. Moreresource-intensive processing may be limited to the relevant extractedfeatures and/or underlying data. Alternatively or additionally, aquality score may be determined for each of the data signals. Datasignals with a quality score that is too low (e.g., indicating a verynoisy environment or low signal to noise ratio (SNR)) may be discardedaltogether. Data signals with a quality score that is sufficiently high(e.g., indicating a relatively high SNR) may in some circumstances beused to the exclusion of other data signals. Data signals with qualityscores between the two extremes may be used together to reduceuncertainty. Such quality scores may be calculated on a per segment orper window basis to, e.g., discard segments or windows that are toonoisy and keep segments or windows that have sufficient SNR.

Feature extraction may be adaptive. For instance, feature extraction maybe personalized, e.g., specifically adapted to each subject. As anexample, feature extraction may learn to recognize and/or extract fromdata signals time-domain and/or frequency-domain features that arespecific to each subject and/or may be more rigorous or extensive forsubjects that belong to certain classes of subjects (e.g., subjects withknown history of a particular condition). Alternatively or additionally,feature extraction may be adaptive to each environment in which subjectsmay be found. For instance, a relatively low resolution featureextraction may be implemented in low noise environments compared to arelatively higher resolution feature extraction in higher noiseenvironments. Alternatively or additionally, feature extraction may beadaptive to a growing knowledge base or data set to update over timealgorithms and/or state machines used to detect sound-producingbehaviors.

In some embodiments, subjects may provide annotation inputs (orannotations). The annotations may be provided by the subjects of theirown accord or in response to a request from the wearable electronicdevice. The annotations may be provided by input into the wearableelectronic device or through other electronic devices. The annotationsmay confirm, or not, the occurrence of one or more sound-producingbehaviors. For instance, if the data signals indicate the occurrence ofa sound-producing behavior such as a sneeze, the wearable electronicdevice may output a message to ask the subject whether the subject justsneezed. The subject can respond with an appropriate input to confirm ordeny the occurrence of the sneeze, which input may be received by thewearable electronic device as an annotation. The annotations may be fedback into the wearable electronic device and/or the cloud (e.g., aremote server), which may update algorithms and/or state machines usedto detect sound-producing behaviors.

Wearable electronic devices as described herein can be used by numeroussubjects. Annotations can be received from some or all of the subjectswith significant network effects. For instance, as more subjects use thewearable electronic devices and more annotations are fed back, e.g., tothe cloud to update the algorithms and/or state machines used to detectsound-producing behaviors, the algorithms and/or state machines maybecome increasingly accurate.

Reference will now be made to the drawings to describe various aspectsof some example embodiments of the disclosure. The drawings arediagrammatic and schematic representations of such example embodiments,and are not limiting of the present disclosure, nor are they necessarilydrawn to scale.

FIG. 1 illustrates an example environment 100 in which some embodimentsdescribed herein can be implemented. The environment 100 includes asubject 102 and a wearable electronic device 104. The environment 100may additionally include a smartphone 106 (or other personal electronicdevice), a cloud computing environment (hereinafter “cloud 108”) thatincludes at least one remote server 110, a network 112, multiple thirdparty user devices 114 (hereinafter “user device 114” or “user devices114”), and multiple third parties (not shown). The user devices 114 mayinclude wearable electronic devices and/or smartphones of other subjectsor users not illustrated in FIG. 1. The environment 100 may include oneor more sensor devices 116, such as the devices 116 a, 116 b, and/or 116c.

The network 112 may include one or more wide area networks (WANs) and/orlocal area networks (LANs) that enable the wearable electronic device104, the smartphone 106, the cloud 108, the remote server 110, thesensor devices 116, and/or the user devices 104 to communicate with eachother. In some embodiments, the network 112 includes the Internet,including a global internetwork formed by logical and physicalconnections between multiple WANs and/or LANs. Alternately oradditionally, the network 112 may include one or more cellular RFnetworks and/or one or more wired and/or wireless networks such as, butnot limited to, 802.xx networks, Bluetooth access points, wirelessaccess points, IP-based networks, or the like. The network 112 may alsoinclude servers that enable one type of network to interface withanother type of network.

The environment 100 additionally includes one or more sensor devices116. Each of the sensors of the sensor devices 116 are configured togenerate data signals that measure parameters that may be indicative ofbehaviors and/or biometric responses of the subject 102. The measuredparameters may include, for example, sound near the subject 102,acceleration of the subject 102 or of a chest, hand, wrist, or otherpart of the subject 102, angular velocity of the subject 102 or of achest, hand, wrist, or other part of the subject 102, temperature of theskin of the subject 102, core body temperature of the subject 102, bloodoxygenation of the subject 102, blood flow of the subject 102,electrical activity of the heart of the subject 102, EDA of the subject102, or other parameters, one or more of which may be indicative ofcertain sound-producing behaviors of the subject 102, such as sneezing,coughing, vomiting, or shouting. The wearable electronic device 104, thesmartphone 106, and/or the remote server 110 may be configured todetermine or extract one or more features from the data signals and/orfrom data derived therefrom to detect sound-producing behaviors of thesubject 102.

All of the sensors may be included in a single device, such as thesensor device 116, the wearable electronic device 104, and/or thesmartphone 106. Alternately or additionally, the sensors may bedistributed between two or more devices. For instance, one or each ofthe sensor devices 116, the wearable electronic device 104 or thesmartphone 106 may include a sensor. Alternately or additionally, theone or more sensors may be provided as separate sensors that areseparate from either of the wearable electronic device 104 or thesmartphone 106. For example, the sensor devices 116 may be provided asseparate sensors. In particular, the sensor devices 116 are separatefrom the wearable electronic device 104 and the smartphone 106, and isembodied in FIG. 1 as a chest-strap style heart rate monitor (e.g.,electrocardiogram (ECG) sensor). In other embodiments, the distributedsensor devices 116 may include any of a discrete microphone,accelerometer, gyro sensor, thermometer, oxygen saturation sensor, PPGsensor, ECG sensor, EDA sensor, or other sensor. In some embodiments,each of the sensor devices 116 may include multiple sensors and multiplesensors devices may be used in analyzing the subject 102. For example, afirst sensor device 116 a may be positioned along a sternum of thesubject 102; as another example, a second sensor device 116 b may bepositioned over the left breast to be over the heart; as an additionalexample, a third sensor device 116 c may be positioned beneath the leftarm of the subject 102. In these and other embodiments, the differentsensor devices 116 at the different locations may be beneficial for amore robust set of data for analyzing the subject 102. For example,different locations of the sensor devices 116 may identify differentfeatures based on their respective locations proximate a chest cavity ofthe subject 102.

In some embodiments, the sensor devices 116 and/or the sensor(s)included in one or more of the wearable electronic device 104 and/or thesmartphone 106 can include a discrete or integrated sensor attached toor otherwise born on the body of the subject 102. Various non-limitingexamples of sensors that may be attached to the body of the subject 102or otherwise implemented according to the embodiments described hereinand that may be implemented as the sensor device 116 and/or as thesensor(s) included in the wearable electronic device 104 or thesmartphone 106 include microphones, PPG sensors, accelerometers, gyrosensors, heart rate sensors (e.g., pulse oximeters), ECG sensors, EDAsensors, or other suitable sensors. In an example implementation, atleast two sensors are provided in an integrated form (e.g., all sensorsincluded in the wearable electronic device 104 or the smartphone 106) orin a distributed form (e.g., at least one sensor in the wearableelectronic device 104 and at least one sensor in the smartphone 106and/or as the sensor device 116). The two sensors may include amicrophone and at least one of an accelerometer, a gyro sensor, a PPGsensor, an ECG sensor, or an EDA sensor.

As already mentioned, the sensors may each be configured to generatedata signals, e.g., of sounds, acceleration, angular velocity, bloodflow, electrical activity of the heart, EDA, or of other parameters ofor near the subject 102. The data signals may be processed to determinea quality score for each and/or to extract features that may beindicative of sound-producing behaviors of the subject 102. Thesound-producing behaviors may include, e.g., sneezing, coughing,vomiting, shouting, or other sound-producing behaviors.

The wearable electronic device 104 may be embodied as a portableelectronic device and may be worn by the subject 102 throughout the dayand/or at other times. The wearable electronic device 104 may beimplemented as a sticker that is stuck to the chest of the subject 102,worn on a wrist of the subject 102 as illustrated in FIG. 1, may becarried in a pocket or clipped to a belt of the subject 102, or may beborn in some other manner by the subject 102. As used herein, “born by”means carried by and/or attached to. The wearable electronic device 104may be configured to, among other things, analyze signals collected byone or more sensors within the environment 100 to measure/detectsound-producing behaviors of the subject 102. In these and otherembodiments, the wearable electronic device 104 may be a devicededicated for performing such functionality. The wearable electronicdevice 104 may include at least one onboard sensor for collecting suchsignals. Alternately or additionally, the smartphone 106 may include atleast one sensor and may communicate signals collected by its onboardsensor to the wearable electronic device 104, and/or the wearableelectronic device 104 may communicate with the sensor device 116 orother separate sensors to receive signals collected by the sensor device116 or other separate sensor(s). The wearable electronic device 104and/or the smartphone 106 may process the data signals collected fromthe sensors to detect one or more sound-producing behaviors of thesubject 102. In these and other embodiments, the wearable electronicdevice 104 and/or the smartphone 106 may detect sound-producingbehaviors by, e.g., executing one or more algorithms and/or statemachines to classify extracted features as being indicative, or not, ofsound-producing behaviors.

The wearable electronic device 104 and/or the smartphone 106 may be usedby the subject 102 to perform journaling, including providing subjectiveannotations to confirm or deny the occurrence of sound-producingbehaviors. Additional details regarding example implementations ofjournaling using a wearable electronic device or other device aredisclosed in copending U.S. application Ser. No. 15/194,145, entitledWEARABLE DEVICE JOURNALING AND COACHING and filed on Jun. 27, 2016,which is incorporated herein by reference in its entirety. The subject102 may provide annotations any time desired by the subject 102, such asafter sneezing, coughing, vomiting, or shouting, and without beingprompted by the wearable electronic device 104 or the smartphone 106.Alternatively or additionally, the subject 102 may provide annotationsresponsive to prompts from the wearable electronic device 104 or thesmartphone 106. For instance, in response to detecting a sound-producingbehavior based on data signals generated by one or more sensors, thewearable electronic device may provide an output to the subject 102 toconfirm whether the detected sound-producing behavior actually occurred.The annotations may be provided to the cloud 108 and in particular tothe remote server 110.

The remote server 110 may include a collection of computing resourcesavailable in the cloud 108. The remote server 110 may be configured toreceive annotations and/or data derived from data signals collected byone or more sensors or other devices, such as the wearable electronicdevice 104 or smartphone 106, within the environment 100. Alternativelyor additionally, the remote server 110 may be configured to receive fromthe sensors relatively small portions of the data signals. For instance,if the wearable electronic device 104 or smartphone 106 detects, withinsufficient certainty, a sound-producing behavior of the subject 102,the wearable electronic device 104 or the smartphone 106 may upload tothe remote server 110 a relatively small snippet of data for furtheranalysis at the remote server 110. For instance, if the data signalincludes an audio recording of the subject 102 and extracted featuresfrom the foregoing and/or other data signals indicate, but withinsufficient certainty, that the subject 102 experienced asound-producing behavior, a 5-10 second snippet of the data signal inwhich the sound-producing behavior appears to have occurred may be sentto the remote server 110. The remote server 110 may then apply morerigorous processing to the snippet to extract additional features and/orhigher resolution features from the snippet than was applied at theconstrained (e.g., power-constrained) wearable electronic device 104and/or smartphone 106. Alternatively or additionally to the furtherprocessing, a live human may listen to the snippet uploaded to theremote sever 110 to determine whether the sound-producing behavioroccurred.

In some embodiments, the wearable electronic device 104 and/orsmartphone 106 may transmit the data signals to the remote server 110such that the remote server may detect the sound-producing behavior.Additionally or alternatively, the wearable electronic device 104 and/orsmartphone 106 may detect the sound-producing behavior from the datasignals locally at the wearable electronic device 104 and/or smartphone106. In these and other embodiments, a determination of whether or notto perform the detection of the sound-producing behavior locally orremotely may be based on capabilities of the processor of the localdevice, power capabilities of the local device, remaining power of thelocal device, communication channels available to transmit data to theremote server 110 (e.g., Wi-Fi, Bluetooth, etc.), payload size (e.g.,how much data is being communicated), cost for transmitting data (e.g.,a cellular connection vs. a Wi-Fi connection), etc. For example, if thewearable electronic device 104 includes a battery as a power source thatis not rechargeable, the wearable electronic device 104 may includesimple sound-producing behavior detection, and otherwise may send thedata signals to the remote server 110 for processing. As anotherexample, if the wearable electronic device 104 includes a rechargeablebattery that is full, the wearable electronic device 104 may perform thedetection locally when the battery is full or close to full and maydecide to perform the detection remotely when the battery has lesscharge. As described in the present disclosure, the detection of thesound-producing behavior may include one or more steps, such as featureextraction, identification, and/or classification. In these and otherembodiments, any of these steps or processes may be performed at anycombination of devices such as at the wearable electronic device 104,the smartphone 106, and/or the remote server 110. For example, thewearable electronic device 104 may collect data and perform someprocessing on the data (e.g., collecting audio data and performing apower spectral density process on the data), provide the processed datato the smartphone 106, and the smartphone 106 may extract one or morefeatures in the processed data, and may communicate the extractedfeatures to the remote server 110 to classify the features into one ormore sound producing behaviors.

In some embodiments, an intermediate device may act as a hub to collectdata from the wearable electronic device 104 and/or the smartphone 106.For example, the hub may collect data over a local communication scheme(Wi-Fi, Bluetooth, near-field communications (NFC), etc.) and maytransmit the data to the remote server 110. In some embodiments, the hubmay act to collect the data and periodically provide the data to theremote server 110, such as once per week.

The remote server 110 may maintain one or more of the algorithms and/orstate machines used in the detection of sound-producing behaviors by thewearable electronic device 104 and/or the smartphone 106. In someembodiments, annotations or other information collected by, e.g., thewearable electronic device 104, the smartphone 106, and/or the userdevices 114, for multiple subjects may be fed back to the cloud 108 toupdate the algorithms and/or state machines. This may advantageouslylead to significant network effects, e.g., as more information iscollected from more subjects, the algorithms and/or state machines usedto detect sound-producing behaviors may be updated to becomeincreasingly accurate and/or efficient. The updated algorithms and/orstate machines may be downloaded from the remote server 110 to thewearable electronic device 104, the smartphone 106, and/or the userdevices 114 to, e.g., improve detection.

FIG. 2 is a block diagram of the wearable electronic device 104 andremote server 110 of FIG. 1, arranged in accordance with at least oneembodiment described herein. Each of the wearable electronic device 104and the remote server 110 may include a processor 202A or 202B(generically “processor 202” or “processors 202”), a communicationinterface 204A or 204B (generically “communication interface 204” or“communication interfaces 204”), and a storage and/or memory 206A or206B (generically “storage 206”). Although not illustrated in FIG. 2,the smartphone 106 (or other personal electronic device) of FIG. 1 maybe configured in a similar manner as the wearable electronic device 104as illustrated in FIG. 2. For instance, the smartphone 106 may includethe same, similar, and/or analogous elements or components asillustrated in FIG. 2.

Each of the processors 202 may include an arithmetic logic unit, amicroprocessor, a general-purpose controller, or some other processor orarray of processors, to perform or control performance of operations asdescribed herein. The processors 202 may be configured to process datasignals and may include various computing architectures including acomplex instruction set computer (CISC) architecture, a reducedinstruction set computer (RISC) architecture, or an architectureimplementing a combination of instruction sets. Although each of thewearable electronic device 104 and the remote server 110 of FIG. 2includes a single processor 202, multiple processor devices may beincluded and other processors and physical configurations may bepossible. The processor 202 may be configured to process any suitablenumber format including, but not limited to two's compliment numbers,integers, fixed binary point numbers, and/or floating point numbers,etc. all of which may be signed or unsigned.

Each of the communication interfaces 204 may be configured to transmitand receive data to and from other devices and/or servers through anetwork bus, such as an I2C serial computer bus, a universalasynchronous receiver/transmitter (UART) based bus, or any othersuitable bus. In some implementations, each of the communicationinterfaces 204 may include a wireless transceiver for exchanging datawith other devices or other communication channels using one or morewireless communication methods, including IEEE 802.11, IEEE 802.16,BLUETOOTH®, Wi-Fi, Zigbee, near field communication (NFC), or anothersuitable wireless communication method.

The storage 206 may include a non-transitory storage medium that storesinstructions or data that may be executed or operated on by acorresponding one of the processors 202. The instructions or data mayinclude programming code that may be executed by a corresponding one ofthe processors 202 to perform or control performance of the operationsdescribed herein. The storage 206 may include a non-volatile memory orsimilar permanent storage media including a flash memory device, anelectrically erasable and programmable read only memory (EEPROM), amagnetic memory device, an optical memory device, or some other massstorage for storing information on a more permanent basis. In someembodiments, the storage 206 may also include volatile memory, such as adynamic random access memory (DRAM) device, a static random accessmemory (SRAM) device, or the like.

The wearable electronic device 104 may additionally include one or moresensors 208, a feature extractor 210A, a classifier 212A, and/or anannotation interface 214. The storage 206A of the wearable electronicdevice 104 may include one or more of data 216, extracted features 218,and/or detected sound-producing behaviors 220 (hereinafter “detectedbehaviors 220”).

The sensor 208 may include one or more of a microphone, anaccelerometer, a gyro sensor, a PPG sensor, an ECG sensor, or an EDAsensor. In embodiments in which the wearable electronic device 104includes or uses a single sensor 208 to detect sound-producing behaviorsof the subject 102, one or more other sensors such as the discretesensor 114 and/or an integrated sensor of the smartphone 106 maycommunicate one or more corresponding data signals to the wearableelectronic device 104 (e.g., through the communication interface 204A).In these and other embodiments, data signals from two or more sensorsmay be processed at a constrained device such as the wearable electronicdevice 104 to detect sound-producing behaviors with greater certaintythan could be done with a single data signal itself. While only a singlesensor 208 is illustrated in FIG. 2, more generally the wearableelectronic device 104 may include one or more sensors.

In some embodiments, the wearable electronic device 104 may includemultiple sensors 208, with a trigger from one sensor 208 causing anothersensor 208 to receive power and start capturing data. For example, anaccelerometer, gyro sensor, EKG, etc. may trigger a microphone to beginreceiving power to capture audio data.

The feature extractor 210A, the classifier 212A, and the annotationinterface 214 may each include code such as computer-readableinstructions that may be executable by the processor 202A of thewearable electronic device 104 to perform or control performance of oneor more methods or operations as described herein. For instance, thefeature extractor 210A may in some embodiments divide data signals intoframes and/or windows and extract one or more features from the datasignals in the time domain and/or the frequency domain. The classifier212A may in some embodiments classify frames and/or windows asindicative of sound-producing behaviors based on the extracted featuresfrom the feature extractor 210A and/or may perform additional and/ormore robust feature extraction on some of the extracted features and/orunderlying data. In some embodiments, the classifier 212A may identifyone or more wake-up events in underlying data that excludes audio data.Based on the identification of such a feature associated with a wake-upevent, the processor 202A may send a signal to cause the microphone toreceive power and begin capturing audio data. The annotation interface214 may in some embodiments prompt the subject 102 for and/or receivefrom the subject 102 annotations. An example method that may beperformed or controlled by execution of one or more of the featureextractor 210A, the classifier 212A, and/or the annotation interface 214is described below with respect to FIG. 4. The feature extractor 210A,the classifier 212A, and the annotation interface 214 may be stored inthe storage 206A or other non-transitory medium.

The data 216 may include some or all of each data signal generated byeach sensor 208. In an example embodiment, portions of each data signalmay be stored temporarily in the storage 206A for processing (e.g.,feature extraction) and may be discarded after processing, to bereplaced by another newly collected portion of the data signal.Alternatively or additionally, one or more portions of one or more datasignals may be retained in storage 206A even after being processed. Insome embodiments, certain sensors may continuously gather data, whileothers may intermittently capture data. For example the data 216 maycontain continuous data from an accelerometer but only a few windows ofdata from a microphone.

In some embodiments, the size of the data 216 stored may be based on thecapacity of the storage 206A. For example, if the storage 206A includeslarge amounts of storage, longer windows of time of the data 216 may bestored, while if the storage 206A includes limited amounts of storage,shorter windows of time of the data 216 may be stored. As anotherexample, if the storage 206A includes large amounts of storage, multipleshort windows of time of the data 216 may be stored, while if thestorage 206A includes limited amounts of storage, a single windows oftime of the data 216 may be stored.

The extracted features 218 may include features extracted by the featureextractor 210A and/or the classifier 212A that may be indicative ofsound-producing behaviors of interest, sound-producing behaviors orother sounds not of interest, and/or undetermined. Examples ofsound-producing behaviors or other sounds not of interest may includethe subject 102 talking, clearing his/her throat, sniffling, or soundsproduced by other people or objects near or around the subject 102. Inan example embodiment, extracted features for sound-producing behaviorsor other sounds that are not of interest may be discarded and featuresindicative of sound-producing behaviors of interest and/or undeterminedmay be retained. Alternatively or additionally, extracted featuresindicative of sound-producing behaviors of interest and/or undeterminedmay be discarded after being further processed and/or classified, e.g.,by the classifier 212A, and/or after being reported to the remote server110.

The detected behaviors 220 may include sound-producing behaviors ofinterest and/or other sound-producing behaviors detected based on one ormore of the extracted features 218. Each of the detected behaviors 220may include a classification of the detected behaviors, a time at whichthe detected behaviors occurred, and/or other information.

In some embodiments, the sensors 208 may include a microphone and atleast one other sensor. The processor 202A may continually monitor thedata 216 from the other sensor other than the microphone (e.g., anaccelerometer). The data 216 from the other sensor may be continuouslygathered and discarded along a running window (e.g., storing a window of10 seconds, discarding the oldest time sample as a new one is obtained).In these and other embodiments, as the feature extractor 210A identifiesa feature for waking up the microphone (e.g., a rapid accelerationpotentially identified as a sneeze), the data 216 may include a windowof audio data. The feature extractor 210A may analyze both the data 216from the other sensor and the data 216 from the microphone to extractone or more features 218.

Referring to the remote server 110, it may additionally include afeature extractor 210B, a classifier 212B, and/or a machine learning(ML) module 222. The storage 206B of the remote server 110 may includeone or more of subject data 224 and/or detection algorithms 226. Thesubject data 224 may include snippets of data, extracted features,detected behaviors, and/or annotations received from wearable electronicdevices and/or smartphones used by subjects, such as the wearableelectronic device 104. The detection algorithms 226 may includealgorithms and/or state machines used by the wearable electronic device104 and/or the remote server 110 in the detection of, e.g.,sound-producing behaviors.

The feature extractor 210B, the classifier 212B, and the ML module 222may each include code such as computer-readable instructions that may beexecutable by the processor 202B of the remote server 110 to perform orcontrol performance of one or more methods or operations as describedherein. For instance, one or both of the feature extractor 210B or theclassifier 212B may perform the same or analogous operations as one orboth of the classifier 212A or feature extractor 210A of the wearableelectronic device 104. For instance, the feature extractor 210B and theclassifier 212B may in some embodiments perform the same and/or morerobust, more processor-intensive, more power-intensive and/or higherresolution processing of snippets of data signals, extracted features,and/or other data received from the wearable electronic device 104 ascompared to the feature extractor 210A and the classifier 212A. The MLmodule 222 may evaluate some or all of the subject data 224 to generateand/or update the detection algorithms 226. For instance, annotationstogether with extracted features and/or detected behaviors or othersubject data 224 may be used as training data by the ML module 222 togenerate and/or update the detection algorithms 226. Updated detectionalgorithms 226 used in feature extraction, classification, or otheraspects of sound-producing behavior detection may then update one ormore of the feature extractors 210A, 210B and/or classifiers 212A, 212Bor other modules in one or both of the remote server 110 and wearableelectronic device 104.

FIGS. 3A-3I include various examples of audio data signal and variousextracted features, arranged in accordance with at least one embodimentdescribed herein. Each of FIGS. 3A-3I illustrate a series of differentsound-producing behaviors in either or both of an audio data plot (300),and a power spectral density (PSD) spectrogram computed using short-timeFourier transform or otherwise processed audio data plot (350). FIG. 3Aillustrates coughing, FIG. 3B illustrates talking, FIG. 3C illustratestalking with music, FIG. 3D illustrates vomiting, FIG. 3E illustratesnose blowing, FIG. 3F illustrates sneezing, FIG. 3G illustratessniffling, FIG. 3H illustrates yelling, and FIG. 3I illustrateslaughing.

As illustrated in FIG. 3A, the audio data plot 300 a may include anamplitude signal 310 a representing the audio data as captured by amicrophone or other comparable sensor. The amplitude signal 310 a mayinclude one or more features 320 a. For example, there may be a featureat approximately 3 seconds, 5 seconds, 8 seconds, 10 seconds, and 13seconds.

In addition to the audio data plot 300 a, FIG. 3A may include aprocessed audio data plot 350 a. The processed audio data plot 350 a mayinclude one or more features 360 a that may correspond to the one ormore features 320 a. For example, the feature 360 a at approximately 13seconds may correspond to the feature 320 a of the amplitude signal 310a at approximately 13 seconds.

In some embodiments, the processed audio data plot 350 a may represent atime-frequency analysis of the amplitude signal 310 a. In someembodiments, such a time-frequency analysis may be one in which theoriginal audio data may not be recreated from the output of thetime-frequency analysis. For example, the processed audio data plot 350a may include a power spectral density (PSD) spectrogram plot. In theseand other embodiments, the processed audio data plot 350 a may representa depiction of the density of power at different frequencies atdifferent times.

In some embodiments, the use of such an approach may facilitate a moresecure and/or more private implementation. For example, the processedaudio data plot 350 a may be one in which the original audio data (e.g.,the amplitude signal 310 a) may not be recreated from the processedaudio data plot 350 a. In these and other embodiments, if there istalking in the original audio data, such a process may provideconfidence that a third party and/or the party providing the device tocapture the audio data will not be listening to conversations of theuser.

In some embodiments, features such as the feature 360 a may beidentified based on pattern matching of the feature 360 a with knownpatterns of any of the sound-producing behaviors (e.g., coughs, sneezes,etc.). Additionally or alternatively, such pattern matching may produceambiguous results, or more than one potential sound-producing behavioras causing the feature, and accelerometer data or any other sensor datamay be included in the feature identification process to facilitate animproved confidence in identification of the feature.

In some embodiments, rather than extracting features from data such ascombinations of audio data and accelerometer data, features may beextracted from representations of the data using a time-frequencyanalysis. For example, the audio data may be converted from the timedomain to the frequency domain over a sliding time window using ashort-time Fourier transform based on Fast Fourier Transform (FFT). Theresulting PSD spectrogram of the audio data may be analyzed to extractfeatures that correlate with behaviors such as sneezing, coughing,vomiting, shouting, etc. In some embodiments, the window of audio datain the PSD spectrogram may be compared with data from one or more othersensors to increase the confidence in the identification of the feature.For example, analysis of the audio data in the PSD spectrogram mayprovide a 65% probability that an event is a cough, and a 25%probability it is a sneeze, and a 10% chance it is neither, while ifdata of an accelerometer is included in the analysis, the probabilitiesmay change to an 80% probability that the event is a cough, and a 12%probability that it is a sneeze, and an 8% probability that it isneither.

In some embodiments, feature detection may be performed in any othermanner, such as that generally described by, e.g., Xiao Sun et al.,SymDetector: Detecting Sound-Related Respiratory Symptoms UsingSmartphones, available athttp://www.cse.psu.edu/˜wwh5068/paper/ubicomp-xiaosun15.pdf (accessed onNov. 10, 2016, hereinafter “Sun”). For instance, the amplitude signal310a may be segmented into non-overlapped frames of 50 milliseconds (ms)for feature extraction, where several continuous frames may be groupedtogether as a window to be processed as a unit. Extracted features mayinclude one or more of root mean square (RMS), above a-mean ratio (AMR),average of top k RMSs (ATR), and/or other features as disclosed by Sun.The various features may be generally used to (1) identify potentialsound-producing behaviors which may include sound-producing behaviors ofinterest, those not of interest, and other discrete sounds not ofinterest, (2) filter out all sounds not of interest, and (3) distinguishthe sound-producing behaviors from each other. For instance, the RMS maybe a measure of energy contained in each acoustic frame and can be usedto identify (and discard) those frames that do not have any potentialsounds of interest. The AMR may be a measure of those windows thatinclude discrete acoustic events, where a low AMR indicates a discreteacoustic event. The other features that may be extracted as disclosed bySun may be used to distinguish sounds that are not of interest accordingto embodiments disclosed herein (such as talking, throat clearing,sniffling, door closing) from sounds that are of interest according toembodiments disclosed herein (e.g., coughing, sneezing, and/or others).

In some embodiments, certain considerations may be involved inimplementing the present disclosure based on a location of the sensorand/or a gender of the subject. For example, if the sensors arephysically attached to the chest of the subject (e.g., via a sticker),the microphone may be oriented to be listening to the chest cavity ofthe subject, rather than oriented to be listening to the ambientsurroundings of the subject, in a similar manner to how a physicianmight orient a stethoscope. In these and other embodiments, the audiodata may be more attuned to certain frequencies and/or patternsdepending on the event of interest. For example, a cough may cause acertain spike in a certain frequency when heard via an ambient-listeningmicrophone, but may include two or more related spike in frequencieswhen heard via a microphone oriented inwardly towards the chest cavityof the subject.

In some embodiments, the gender of the subject may modify or otherwiseadjust the patterns and/or frequencies observed when identifying variousevents. For example, for men, pectoral muscles may impede or alter thefrequencies observed for various events. As another example, breasttissue for females may impede or alter the frequencies observed forvarious events. In light of such variations, events may be identified,the microphone may be triggered to wake up to begin capturing audio,etc. based on a gender of the subject.

In some embodiments, the orientation of the microphone may yieldincreased insight into various aspects of the event. For example, if anevent is identified as a cough, by having the microphone orientedagainst the chest of the subject, the audio data collected by themicrophone may be able to detect whether there are any particularaspects of the event of note, such as any rattles or whistles associatedwith the cough, whether or not there is any buildup of mucus, phlegm, orother constriction of airways heard in the cough, etc.

According to embodiments described herein, data signals collected inpractice may be relatively noisy. Thus, multiple data signals (e.g.,audio, acceleration, angular velocity, blood flow, ECG, and/or EDA datasignals) may be collected to extract features from each that may reduceuncertainty in the detection of sound-producing behaviors. For instance,if a first data signal is somewhat noisy at a particular time such thata feature or features extracted from the first data signal suggest theoccurrence of a sound-producing behavior but are somewhat ambiguous orunclear, a feature or features extracted from a second data signal forthe same particular time that also suggest or even more clearly indicatethe occurrence of the sound-producing behavior may reduce theuncertainty by considering features from both data signals.Alternatively or additionally, uncertainty may be reduced by consideringfeatures from three or more data signals.

In still other implementations, different sound-producing behaviors mayhave similar characteristics or features in a given data signal even ifthe given data signal is relatively clean, but may have more distinctcharacteristics or features in a different data signal that can be usedto discriminate between the two. For instance, sneezing and coughing maybe somewhat similar in duration and amplitude in an audio data signaleven if the audio data signal is relatively clean. To discriminatebetween the two to detect whether a sound-producing behavior is a sneezeor a cough, a different data signal may be used that may have lessambiguity in characteristics and/or features between sneezing andcoughing.

Accordingly, uncertainty in detecting sound-producing behaviors can bereduced by using data signals from two or more different sensors thatgenerate data signals relating to different parameters. By using atleast two different sensors that detect different parameters, noise orambiguity that may be present in one of the data signals may be absentfrom the other.

Embodiments described herein may extract features from data signals,where the features may be indicative of a sound-producing behavior. Forinstance, as illustrated by the amplitude signal 310 a in FIG. 3A,sound-producing behaviors may be characterized by a spike in amplitudein the amplitude signal 310 a, a spike in the processed audio data plot350 a, and/or by other characteristics and/or features. Alternatively oradditionally, the particular sound-producing behaviors of interest maybe characterized by certain movements or physiological responses thatmay be measurable by other sensors as described herein. Featuresindicative of such characteristics may be extracted from a correspondingdata signal. Examples of such features may include one or more of rapidacceleration of the chest during a cough, a lower and more prolongedacceleration during vomiting, etc.

In an example implementation, feature extraction and classification mayoccur substantially as follows. Audio data or other data may beidentified as including a feature. For example, as illustrated in FIG.3A, the spike in amplitude around 13 seconds may indicate that a featuremay have occurred at that time. The associated audio data (or all theaudio data) may be processed for time-frequency analysis (e.g., a powerspectral density (PSD) spectrogram). Additionally or alternatively, theprocessing may include filtering or other analysis. In some embodiments,features may be extracted by processing all audio data within a timewindow after a wake-up event experienced by an accelerometer. Afterextracting one or more features, a pattern matching process may beperformed on the extracted features. For example, the PSD spectrogram ofthe extracted feature may be compared to multiple reference PSDspectrograms of known sound-producing behaviors. In some embodiments,these reference PSD spectrograms may be applicable to any number ofusers. Additionally or alternatively, the reference PSD spectrograms maybe generated during an enrollment session for a given user (e.g., whenreceiving a wearable device with sensors, the user may be asked tocough, blow their nose, read a passage, etc. and the associated sensorresponses for such sound-producing behaviors may be collected andstored, including generation of PSD spectrograms on the audio and/orother data associated with the reference sound-producing behaviors). Inaddition or alternatively to a pattern matching process, any othercategorization process, such as a machine learning process (e.g.,logistic regression, decision trees, neural networks, etc.) may be usedto identify to which category of sound-producing behavior the PSDspectrogram belongs.

As illustrated in FIG. 3B, the audio data plot 300 b may include theamplitude signal 310 b of a user talking. The amplitude signal 310 b mayinclude one or more features 320 b. For example, there may beintermittent features between approximately 5 seconds and 14 seconds,and approximately 15 seconds.

In addition to the audio data plot 300 b, FIG. 3B may include aprocessed audio data plot 350 b. The processed audio data plot 350 b mayinclude one or more features 360 b that may correspond to the one ormore features 320 b. For example, the feature 360 b at approximately 5-7seconds may correspond to the feature 320 b of the amplitude signal 310b at approximately 5-7 seconds.

As illustrated in FIG. 3C, the audio data plot 300 c may include theamplitude signal 310 b of a user talking with music. The amplitudesignal 310 c may include one or more features 320 c. For example, theremay be intermittent features between approximately 6 seconds and 8seconds, and approximately 9 seconds and 15 seconds.

In addition to the audio data plot 300 c, FIG. 3C may include aprocessed audio data plot 350 c. The processed audio data plot 350 c mayinclude one or more features 360 c that may correspond to the one ormore features 320 c. For example, the feature 360 c at approximately9-10 seconds may correspond to the feature 320 c of the amplitude signal310 c at approximately 9-10 seconds.

As illustrated in FIG. 3D the audio data plot 300 d may include theamplitude signal 310 d of a user vomiting. The amplitude signal 310 dmay include one or more features 320 d. For example, there may beintermittent features at approximately 4 seconds, 6 seconds, 9 seconds,12 seconds, and 15 seconds.

In addition to the audio data plot 300 d, FIG. 3D may include aprocessed audio data plot 350 d. The processed audio data plot 350 d mayinclude one or more features 360 d that may correspond to the one ormore features 320 d. For example, the feature 360 d at approximately 9seconds may correspond to the feature 320 d of the amplitude signal 310d at approximately 9 seconds.

As illustrated in FIG. 3E the audio data plot 300 e may include theamplitude signal 310 e of a user blowing his nose. The amplitude signal310 e may include one or more features 320 e. For example, there may beintermittent features at approximately 3 seconds, 5 seconds, 8 seconds,11 seconds, and 15 seconds.

In addition to the audio data plot 300 e, FIG. 3E may include aprocessed audio data plot 350 e. The processed audio data plot 350 e mayinclude one or more features 360 e that may correspond to the one ormore features 320 e. For example, the feature 360 e at approximately 8seconds may correspond to the feature 320 e of the amplitude signal 310e at approximately 8 seconds.

As illustrated in FIG. 3F the audio data plot 300 f may include theamplitude signal 310 f of a user sneezing. The amplitude signal 310 fmay include one or more features 320 f. For example, there may beintermittent features at approximately 3 seconds, 5 seconds, 8 seconds,10 seconds, and 13 seconds.

In addition to the audio data plot 300 f, FIG. 3F may include aprocessed audio data plot 350 f. The processed audio data plot 350 f mayinclude one or more features 360 f that may correspond to the one ormore features 320 f. For example, the feature 360 f at approximately 10seconds may correspond to the feature 320 f of the amplitude signal 310f at approximately 10 seconds.

As illustrated in FIG. 3G the audio data plot 300 g may include theamplitude signal 310 g of a user sniffling. The amplitude signal 310 gmay include one or more features 320 g. For example, there may beintermittent features at approximately 3 seconds, 5 seconds, 7 seconds,9 seconds, and 11 seconds.

In addition to the audio data plot 300 g, FIG. 3G may include aprocessed audio data plot 350 g. The processed audio data plot 350 g mayinclude one or more features 360 g that may correspond to the one ormore features 320 g. For example, the feature 360 g at approximately 9seconds may correspond to the feature 320 g of the amplitude signal 310g at approximately 9 seconds.

As illustrated in FIG. 3H the audio data plot 300 h may include theamplitude signal 310 h of a user yelling. The amplitude signal 310 h mayinclude one or more features 320 h. For example, there may beintermittent features at approximately 6-8 seconds, 11-12 seconds, and13-15 seconds.

In addition to the audio data plot 300 h, FIG. 3H may include aprocessed audio data plot 350 h. The processed audio data plot 350 h mayinclude one or more features 360 h that may correspond to the one ormore features 320 h. For example, the feature 360 h at approximately 11seconds may correspond to the feature 320 h of the amplitude signal 310h at approximately 11 seconds.

As illustrated in FIG. 3I the audio data plot 300 i may include theamplitude signal 310 i of a user laughing. The amplitude signal 310 imay include one or more features 320 i. For example, there may beintermittent features at approximately 3-6 seconds, 9 seconds, and 13seconds.

In addition to the audio data plot 300 i, FIG. 3I may include aprocessed audio data plot 350 i. The processed audio data plot 350 i mayinclude one or more features 360 i that may correspond to the one ormore features 320 i. For example, the feature 360 i at approximately 9seconds may correspond to the feature 320 i of the amplitude signal 310i at approximately 9 seconds.

The examples of FIGS. 3A-3I are simply examples and it is appreciatedthat the present disclosure extends to other sound-producing behaviorssuch as wheezing, shortness of breath, sex, masturbation, etc.Additionally, the description related to FIG. 3A regarding featureextraction, frequency analysis, behavior identification, etc. areequally applicable to FIGS. 3B-3I.

FIG. 4 is a flowchart of an example method 400 to measuresound-producing behaviors of a subject with a power- andbandwidth-limited electronic device that includes a processor, arrangedin accordance with at least one embodiment described herein. The method400 may be implemented, in whole or in part, by the wearable electronicdevice 104, the smartphone 106, and/or the remote server 110 describedelsewhere herein. Alternatively or additionally, execution of one ormore of the feature extractor 210A and/or 210B, classifier 212A and/or212B, annotation interface 214, and/or ML module 222 by the processor202A and/or 202B of the wearable electronic device 104 and/or the remoteserver 110 may cause the corresponding processor 202A and/or 202B toperform or control performance of one or more of the operations orblocks of the method 400. The method 400 may include one or more ofblocks 402, 404, and/or 406. The method 400 may begin at block 402.

At block 402, sound may be measured in a vicinity of a subject togenerate an audio data signal that represents the sound. The sound maybe measured by, e.g., a microphone communicatively coupled to theprocessor 202A of FIG. 2. The microphone may be included as one of thesensors 208 of the wearable electronic device 104 or as a discretesensor or as a sensor integrated in a smartphone or other device apartfrom the wearable electronic device 104. The audio data signal maycapture sounds the subject makes, such as coughing, sneezing, vomiting,shouting, talking, moving him/herself or other objects, opening closingdoors, etc., as well as sounds made by others. Sound-producing behaviorsof interest may generally each have a “signature” in the audio datasignal, or recognizable aspects or patterns, that may be identifiedduring feature extraction and/or classification to detectsound-producing behaviors. Block 402 may be followed by block 404.

At block 404, at least one parameter other than sound may be measured togenerate at least a second data signal that represents the at least oneparameter other than sound. The at least one parameter other than soundmay be measured by a corresponding sensor communicatively coupled to theprocessor 202A of FIG. 2. For instance, an accelerometer communicativelycoupled to the processor may measure acceleration of at least a portionof the subject to generate an acceleration data signal that representsacceleration of the at least the portion of the subject. As anotherexample, a gyro sensor communicatively coupled to the processor maymeasure angular velocity of the at least the portion of the subject togenerate an angular velocity data signal that represents angularvelocity of the at least the portion of the subject. As another example,a PPG sensor communicatively coupled to the processor may measure bloodflow of the subject to generate a blood flow data signal that representsblood flow of the subject. As another example, an ECG sensorcommunicatively coupled to the processor, electrical activity of a heartof the subject to generate an ECG data signal that represents electricalactivity of the heart. As another example, an EDA sensor communicativelycoupled to the processor may measure EDA of the subject to generate anEDA data signal that represents EDA of the subject.

Sound-producing behaviors and/or behaviors related thereto may generallyhave a signature in the corresponding data signal that may be identifiedduring feature extraction and/or classification to detectsound-producing behaviors or the related behaviors. For example vomitingand/or shouting associated with rage (or mood) may be accompanied by anelevated heart rate that may be detected from the blood flow data signaland/or the ECG data signal and/or by changes in EDA that may be detectedfrom the EDA data signal. As another example, sneezing and/or coughingmay be accompanied by particular movements of the subject's arm or otherbody part (e.g., blood within a vessel) that may be detected from theacceleration data signal, the angular velocity data signal, and/or theblood flow data signal. Block 404 may be followed by block 406.

At block 406, one or more sound-producing behaviors of the subject maybe detected based on: both the audio data signal and the second datasignal (e.g., one or more of the acceleration data signal, the angularvelocity data signal, the blood flow signal, the ECG signal, and/or theEDA signal); or information derived from both the audio data signal andthe second data signal. The sound-producing behaviors may be detectedby, e.g., the processor 202A of the wearable electronic device 104and/or the processor 202B of the remote server 110 of FIG. 2.

The detecting may include detecting a sound-producing behavior at timeswhen both the audio data signal and the second data signal or theinformation derived therefrom indicates a sound-producing behavior.Alternatively or additionally, the detecting may include extractingaudio features from the audio data signal; extracting second featuresfrom the second data signal; classifying each audio feature of a firstsubset of the audio features and each feature of a first subset of thesecond features as indicative of a sound-producing behavior; anddetecting the one or more sound-producing behaviors of the subject basedon the classifying. The detecting may alternatively or additionallyinclude recording each detected sound-producing behavior as such,including recording a time at which each sound-producing behavioroccurred and/or other information about each sound-producing behavior.The detecting may further include sub-classifying or categorizing eachaudio feature of the first subset of audio features and each feature ofthe first subset of second features as indicative of a correspondingtype of multiple different types of sound-producing behaviors (e.g.,cough, sneeze, vomit, shouting)

Although not illustrated in FIG. 4, the method 400 may include one ormore other operations instead of or in addition to those illustrated inFIG. 4. For instance, the method 400 may further include reportinginformation about sound-producing behaviors of the subject derived fromone or both of the audio data signal or the second data signal to theremote server. For instance, one or more of the audio features extractedfrom the audio data signal may be reported to the remote server.Alternatively or additionally, one or more of the second featuresextracted from the second data signal may be reported to the remoteserver. Alternatively or additionally, one or more detectedsound-producing behaviors and an occurrence time of each of the one ormore detected sound-producing behaviors may be reported to the remoteserver. One or more of the foregoing may be reported to the remoteserver all without reporting to the remote server any of the audio datasignal or any of the second data signal.

In some embodiments, the audio data signal and the second data signalmay be time synchronized with each other. Alternatively or additionally,the audio features and the second features may be time synchronized witheach other. In these and other embodiments, the detecting ofsound-producing behaviors may include, for at least one of thesound-producing behaviors, determining that an audio feature of thefirst subset of audio features coincides chronologically with a secondfeature of the first subset of second features, each of the audiofeature and the second feature being indicative of the sound-producingbehavior.

Alternatively or additionally, where the audio data signal is relativelyclean, sound-producing behaviors may be detected exclusively from theaudio data signal without reference to the second data signal or wherethe audio data signal is very noisy, it may be discarded. For instance,the method 400 may include measuring sound in the vicinity of thesubject within a subsequent period of time to generate a subsequentaudio data signal that represents the sound. The method 400 may includeextracting subsequent audio features from the subsequent audio datasignal. The method 400 may include classifying each audio feature of afirst subset of the subsequent audio features as indicative of asound-producing behavior. The method 400 may include determining aquality score of the subsequent audio data signal. Additionally oralternatively, a quality score may be determined for other data signals.For example, for an audio data signal, the quality score may be based onamplitude peaks and frequencies that are consistent with what isexpected for the particular user or for humans in general. As anotherexample, for the audio data signal, the quality score may be based on asignal to noise ratio. As an additional example, for accelerometer dataover a span of time, the quality score may be based on the shape of acurve of the acceleration data over the span of time, ratios inpeaks/valleys of the acceleration data, frequency ranges of theacceleration data within typical ranges of human physiology, etc. Insome embodiments, such a quality score may be determined based oninitially obtained sensor data, such as initial audio data. Additionallyor alternatively, the quality score may be determined based on processeddata, such as a PSD of audio data. The method 400 may include one of thefollowing. In response to determining that the quality score of thesubsequent audio data signal exceeds a first threshold quality score,the method 400 may include detecting one or more sound-producingbehaviors of the subject that occur within the subsequent period of timebased exclusively on the first subset of the subsequent plurality ofaudio features without considering other data signals and/or featuresextracted from such other data signals. Alternatively, in response todetermining that the quality score of the subsequent audio data signalis less than a second threshold quality score that is lower than thefirst threshold quality score, the method 400 may include discarding oneor both of the subsequent audio data signal and the first subset of thesubsequent audio features without detecting any sound-producingbehaviors of the subject that occur within the subsequent period of timebased thereon.

The method 400 may alternatively or additionally include receiving, fromthe subject, annotation input that may confirm, or not, the occurrenceof one or more sound-producing behaviors. The annotations may be used todecrease measurement uncertainty in the data signals. For instance, ifthe annotations confirm that the subject sneezed, coughed, vomited, orshouted, this increases the certainty that a corresponding detectedsneeze, cough, vomiting, or shout actually occurred. As another example,if the annotations indicate the subject did not actually sneeze, cough,vomit or shout despite the detection of such sound-generating behaviorsby the wearable electronic device, this may indicate that the algorithmsand/or state machines used in the detection of sound-producing behaviorsare inaccurate and should be revised.

Some embodiments described herein may be applied across numeroussubjects. In these and other embodiments, annotations may be requestedfrom some or all of the subjects. For instance, annotations may berequested, e.g., by the wearable electronic device or by a smartphone orother user device, from only a certain percentage of subjects, from onlythose subjects that have opted in to providing annotations, or from allsubjects. Alternatively or additionally, annotations may be requestedfrom a given subject only infrequently, such as only once every X numberof days, where X is an integer greater than 1. Alternatively oradditionally, subjects may provide annotations as frequently as desiredon their own rather than in response to a request for the annotationsfrom the wearable electronic device.

The method 400 may alternatively or additionally include reportingannotations to the remote server. The annotations may be reported to theremote server by the wearable electronic device, or by a smartphone orother user device, that receives the annotations from the subject. Themethod 400 may include the remote server receiving the annotations andinformation about sound-producing behaviors from multiple subjects andupdating one or more algorithms or state machines used in the detection(e.g., extraction and/or classifying) of sound-producing behaviors. Themethod 400 may also include the remote server sending one or moreupdated algorithms or state machines back down to the wearableelectronic device for use in the future in the detection ofsound-producing behaviors.

FIG. 5 is a flowchart of another example method 500 to measuresound-producing behaviors of a subject, arranged in accordance with atleast one embodiment described herein. The method 500 may beimplemented, in whole or in part, by the wearable electronic device 104,the smartphone 106, and/or the remote server 110 described elsewhereherein. Alternatively or additionally, execution of one or more of thefeature extractor 210A and/or 210B, classifier 212A and/or 212B,annotation interface 214, and/or ML module 222 by the processor 202Aand/or 202B of the wearable electronic device 104 and/or the remoteserver 110 may cause the corresponding processor 202A and/or 202B toperform or control performance of one or more of the operations orblocks of the method 500. The method 500 may include one or more ofblocks 502, 504, 506, and/or 508. The method 500 may begin at block 502.

At block 502, at least one parameter other than sound may be measured togenerate a data signal of the parameter and detect a wake-up event fromthe data. The at least one parameter other than sound may be measured bya corresponding sensor communicatively coupled to the processor 202A ofFIG. 2. For instance, an accelerometer communicatively coupled to theprocessor may measure acceleration of at least a portion of the subject(e.g., the chest of the subject) to generate an acceleration data signalthat represents acceleration of the at least the portion of the subject.As another example, a gyro sensor communicatively coupled to theprocessor may measure angular velocity of the at least the portion ofthe subject to generate an angular velocity data signal that representsangular velocity of the at least the portion of the subject. As anotherexample, a PPG sensor communicatively coupled to the processor maymeasure blood flow of the subject to generate a blood flow data signalthat represents blood flow of the subject. As another example, an ECGsensor communicatively coupled to the processor, electrical activity ofa heart of the subject to generate an ECG data signal that representselectrical activity of the heart. As another example, an EDA sensorcommunicatively coupled to the processor may measure EDA of the subjectto generate an EDA data signal that represents EDA of the subject.

In addition to generating the data signal, the data may be analyzed todetermine whether or not a wake-up event has been detected within thedata. For example, the data of an accelerometer may be analyzed todetermine whether a sudden acceleration of the at least the portion ofthe subject (e.g., the chest of the subject) typical of a sneeze orcough has been detected. In some embodiments, a sensor measuring theparameter other than sound may continuously monitor for such a wake-upevent. In these and other embodiments, the threshold of the wake-upevent may be tuned to be more or less sensitive. For example, if acertain number of events is expected in a given twenty four hour periodand less than the amount expected have occurred, the threshold may belowered such that more wake-up events may be triggered. Additionally oralternatively, the threshold may be tuned based on which sound-producingbehavior is being targeted. As another example, if another sensor (e.g.,a temperature sensor) indicates that the subject is experiencing a feveror has an infection, the threshold of the wake-up event may be loweredsuch that more wake-up events are triggered. The block 502 may befollowed by the block 504.

At block 504, a microphone may be woken up based on the wake-up eventsuch that the microphone receives power to measure sound of a subject togenerate an audio data signal that represents the sound. In someembodiments, the microphone may be woken up within one second (forexample, between two hundred and five hundred milliseconds) of thewake-up event. The microphone may be communicatively coupled to theprocessor 202A of FIG. 2 such that the processor sends a signal toprovide power to the microphone. Upon receiving power, the microphonemay begin to capture audio data.

The microphone may be included as one of the sensors 208 of the wearableelectronic device 104 or as a discrete sensor or as a sensor integratedin a smartphone or other device apart from the wearable electronicdevice 104. In some embodiments, the microphone may be oriented directlyagainst the chest of the subject. The audio data signal may capturesounds the subject makes, such as coughing, sneezing, vomiting,shouting, talking, moving him/herself or other objects, opening closingdoors, etc., as well as sounds made by others. The block 504 may befollowed by the block 506.

At block 506, there may be a cessation of power being sent to themicrophone, while the parameter other than sound may continue to bemeasured. In some embodiments, the processor 202A may be configured tocontinue to provide power to the microphone as long as the audio datasignal appears to be associated with an event of interest, and after theaudio data signal no longer appears to be associated with an event ofinterest, the processor may cease to provide the microphone power.Additionally or alternatively, the processor 202A may be configured toprovide power to the microphone for a certain window of time (e.g., twoseconds, four seconds, etc.) based on the wake-up event. In someembodiments, the processor 202A may provide power to the microphonebased on a combination of the foregoing, for example, by providing poweruntil the audio data signal no longer appears to be associated with anevent of interest, after which the microphone may be provided with powerto capture audio data for a certain window of time. In these and otherembodiments, the sensor measuring the parameter other than sound maycontinue to monitor the parameter even if the microphone is notreceiving power.

In some embodiments, the window of time may be based on a talkingdetection algorithm used on the audio data signal. For example, iftalking is detected in the audio data, the window may be extended suchthat additional audio data may be collected. In these and otherembodiments, processing may be performed on the extended window of audiodata and the processed audio data may be retained and the unprocessedaudio data may be discarded. In some embodiments, the lengthened windowof time may provide additional data such that any words spoken in theaudio data cannot be recreated, derived, or otherwise obtained from theprocessed audio data. For example, by including a longer segment thesample size of frequencies and/or powers is extended such that anysingle word is masked by the other words spoken or by other datacollected. Doing so may protect the privacy of the subject and of othersspeaking when around the subject. If talking is not detected, the windowmay be kept to its normal length or even shortened, and the audio datamay be retained or otherwise stored to be used for sound-producingbehavior detection.

Sound-producing behaviors of interest may generally each have a“signature” in the audio data signal (including a frequencyrepresentation thereof), or recognizable aspects or patterns, that maybe identified during feature extraction and/or classification to detectsound-producing behaviors. In some embodiments, such analyses may beperformed on audio data, on processed audio data (e.g., the frequencydomain of the audio data), etc.

Sound-producing behaviors and/or behaviors related thereto may generallyhave a signature in the corresponding data signal that may be identifiedduring feature extraction and/or classification to detectsound-producing behaviors or the related behaviors. For example vomitingand/or shouting associated with rage (or mood) may be accompanied by anelevated heart rate that may be detected from the blood flow data signaland/or the ECG data signal and/or by changes in EDA that may be detectedfrom the EDA data signal. As another example, sneezing and/or coughingmay be accompanied by particular movements of the subject's chest orother body part (e.g., blood within a vessel) that may be detected fromthe acceleration data signal, the angular velocity data signal, and/orthe blood flow data signal. Block 506 may be followed by block 508.

At block 508, one or more sound-producing behaviors of the subject maybe detected based on both the parameter other than sound (e.g., one ormore of the acceleration data signal, the angular velocity data signal,the blood flow signal, the ECG signal, and/or the EDA signal) and thesound; or information derived from both the audio data signal and thesecond data signal. The block 508 may be similar or comparable to theblock 406 of FIG. 4.

Additionally, the method 500 may include any of the additional and notillustrated operations described above with reference to the method 400of FIG. 4.

One skilled in the art will appreciate that, for this and otherprocesses and methods disclosed herein, the functions performed in theprocesses and methods may be implemented in differing order.Furthermore, the outlined steps and operations are only provided asexamples, and some of the steps and operations may be optional, combinedinto fewer steps and operations, or expanded into additional steps andoperations without detracting from the essence of the disclosedembodiments.

The present disclosure is not to be limited in terms of the particularembodiments described herein, which are intended as illustrations ofvarious aspects. Many modifications and variations can be made withoutdeparting from its spirit and scope, as will be apparent to thoseskilled in the art. Functionally equivalent methods and apparatuseswithin the scope of the disclosure, in addition to those enumeratedherein, will be apparent to those skilled in the art from the foregoingdescriptions. Such modifications and variations are intended to fallwithin the scope of the appended claims. The present disclosure is to belimited only by the terms of the appended claims, along with the fullscope of equivalents to which such claims are entitled. It is to beunderstood that the present disclosure is not limited to particularmethods, reagents, compounds, compositions, or biological systems, whichcan, of course, vary. It is also to be understood that the terminologyused herein is for the purpose of describing particular embodimentsonly, and is not intended to be limiting.

With respect to the use of substantially any plural and/or singularterms herein, those having skill in the art can translate from theplural to the singular and/or from the singular to the plural as isappropriate to the context and/or application. The varioussingular/plural permutations may be expressly set forth herein for sakeof clarity.

It will be understood by those within the art that, in general, termsused herein, and especially in the appended claims (e.g., bodies of theappended claims) are generally intended as “open” terms (e.g., the term“including” should be interpreted as “including but not limited to,” theterm “having” should be interpreted as “having at least,” the term“includes” should be interpreted as “includes but is not limited to,”etc.). It will be further understood by those within the art that if aspecific number of an introduced claim recitation is intended, such anintent will be explicitly recited in the claim, and in the absence ofsuch recitation no such intent is present. For example, as an aid tounderstanding, the following appended claims may contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimrecitations. However, the use of such phrases should not be construed toimply that the introduction of a claim recitation by the indefinitearticles “a” or “an” limits any particular claim containing suchintroduced claim recitation to embodiments containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should be interpreted to mean “at least one”or “one or more”); the same holds true for the use of definite articlesused to introduce claim recitations. In addition, even if a specificnumber of an introduced claim recitation is explicitly recited, thoseskilled in the art will recognize that such recitation should beinterpreted to mean at least the recited number (e.g., the barerecitation of “two recitations,” without other modifiers, means at leasttwo recitations, or two or more recitations). Furthermore, in thoseinstances where a convention analogous to “at least one of A, B, and C,etc.” is used, in general such a construction is intended in the senseone having skill in the art would understand the convention (e.g., “asystem having at least one of A, B, and C” would include but not belimited to systems that have A alone, B alone, C alone, A and Btogether, A and C together, B and C together, and/or A, B, and Ctogether, etc.). In those instances where a convention analogous to “atleast one of A, B, or C, etc.” is used, in general such a constructionis intended in the sense one having skill in the art would understandthe convention (e.g., “a system having at least one of A, B, or C” wouldinclude but not be limited to systems that have A alone, B alone, Calone, A and B together, A and C together, B and C together, and/or A,B, and C together, etc.). It will be further understood by those withinthe art that virtually any disjunctive word and/or phrase presenting twoor more alternative terms, whether in the description, claims, ordrawings, should be understood to contemplate the possibilities ofincluding one of the terms, either of the terms, or both terms. Forexample, the phrase “A or B” will be understood to include thepossibilities of “A” or “B” or “A and B.”

The present disclosure may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the disclosure is, therefore,indicated by the appended claims rather than by the foregoingdescription. All changes which come within the meaning and range ofequivalency of the claims are to be embraced within their scope.

1. (canceled)
 2. A method to measure sound-producing behaviors of asubject with a power-and bandwidth-limited electronic device thatincludes a processor, the method comprising: measuring, by at least onesensor communicatively coupled to the processor, at least one parameterother than sound to generate a second data signal that represents the atleast one parameter other than sound; detecting a first wake-up eventoccurring within the second data signal; in response to the detectedfirst wake-up event, providing power to a microphone communicativelycoupled to the processor for a first fixed window of time such that themicrophone measures sound in a vicinity of the subject to generate afirst audio data signal during the first fixed window of time; detectinga second wake-up event occurring within the second data signal; inresponse to the detected second wake-up event, providing power to themicrophone for a second fixed window of time such that the microphonemeasures sound in the vicinity of the subject to generate a second audiodata signal during the second fixed window of time; based on a signalquality of the first audio data signal being below threshold, discardingthe first audio data signal; and detecting one or more sound-producingbehaviors of the subject based on: both the second audio data signal andsecond data signal; or information derived from both the second audiodata signal and the second data signal.
 3. The method of claim 2,wherein measuring, by the at least one sensor, the at least oneparameter other than sound to generate the second data signal comprisesat least one of: measuring, by an accelerometer communicatively coupledto the processor, acceleration of a chest of the subject to generate anacceleration data signal that represents acceleration of the chest ofthe subject; measuring, by a gyro sensor communicatively coupled to theprocessor, angular velocity of at least a portion of the subject togenerate an angular velocity data signal that represents angularvelocity of the at least the portion of the subject; measuring, by athermometer communicatively coupled to the processor, at least one of askin temperature or a core body temperature of the subject to generate atemperature signal that represents at least one of the skin temperatureor the core body temperature of the subject; measuring, by an oxygensaturation sensor communicatively coupled to the processor, bloodoxygenation of the subject to generate an oxygen saturation signal ofthe oxygen saturation level of the subject; measuring, by aphotoplethysmograph (PPG) sensor communicatively coupled to theprocessor, blood flow of the subject to generate a blood flow datasignal that represents blood flow of the subject; measuring, by anelectrocardiograph (ECG) sensor communicatively coupled to theprocessor, electrical activity of a heart of the subject to generate anECG data signal that represents electrical activity of the heart; ormeasuring, by an electrodermal activity (EDA) sensor communicativelycoupled to the processor, EDA of the subject to generate an EDA datasignal that represents EDA of the subject.
 4. The method of claim 2,wherein the detecting further includes: extracting, by the processor, aplurality of audio features from the second audio data signal; andsub-classifying each audio feature of the plurality of audio features asindicative of a corresponding type of sound-producing behavior.
 5. Themethod of claim 4, wherein the corresponding types of sound-producingbehaviors include at least one of sneezing, wheezing, shortness ofbreath, chewing, swallowing, masturbation, sex, coughing, vomiting, andshouting.
 6. The method of claim 2, further comprising: receiving, fromthe subject, annotation input that confirms occurrence of one or moresound-producing behaviors; reporting, to a remote server, the annotationinput, wherein the remote server is configured to receive annotationinputs and information about sound-producing behaviors from a pluralityof subjects and to update an algorithm used in the detecting; andreceiving, from the remote server, the updated algorithm.
 7. The methodof claim 2, wherein the detecting is based on the information derivedfrom both the second audio data signal and the second data signal, themethod further comprising: deriving the information from both the secondaudio data signal and the second data signal by converting the secondaudio data signal from a time domain into a frequency domain across aplurality of frames; and discarding one or more frames withoutidentified sounds of interest.
 8. The method of claim 2, wherein themicrophone is oriented against a chest of the subject.
 9. The method ofclaim 2, wherein the detecting is based on a third data signal from athird sensor monitoring an additional characteristic other than sound.10. The method of claim 2, further comprising generating a referencesignal for a plurality of sound-producing behaviors during an enrollmentprocess.
 11. The method of claim 2, wherein the detecting includesidentifying a plurality of candidate sound-producing behaviors.
 12. Apower-and bandwidth-limited electronic device, comprising: a processor;a first sensor configured to monitor a parameter other than soundcoupled to the processor; a microphone coupled to the processor; acontrollable power source; and a non-transitory storage media containinginstructions that, when executed by the processor, cause the electronicdevice to perform operations, the operations comprising: receiving, fromthe first sensor, a second data signal that represents the parameterother than sound; detecting a first wake-up event occurring within thesecond data signal; in response to the first detected wake-up event,causing the controllable power source to provide power to the microphonefor a first fixed window of time such that the microphone measures soundin a vicinity of a subject to generate a first audio data signal duringthe first fixed window of time; detecting a second wake-up eventoccurring within the second data signal; in response to the detectedsecond wake-up event, causing the controllable power source to providepower to the microphone for a second fixed window of time such that themicrophone measures sound in the vicinity of the subject to generate asecond audio data signal during the second fixed window of time; basedon a signal quality of the first audio data signal being belowthreshold, discarding the first audio data signal; and detecting one ormore sound-producing behaviors of the subject based on: both the secondaudio data signal and second data signal; or information derived fromboth the second audio data signal and the second data signal.
 13. Theelectronic device of claim 12, wherein receiving, from the first sensor,the second data signal comprises at least one of: measuring, by anaccelerometer communicatively coupled to the processor, acceleration ofa chest of the subject to generate an acceleration data signal thatrepresents acceleration of the chest of the subject; measuring, by agyro sensor communicatively coupled to the processor, angular velocityof at least a portion of the subject to generate an angular velocitydata signal that represents angular velocity of the at least the portionof the subject; measuring, by a thermometer communicatively coupled tothe processor, at least one of a skin temperature or a core bodytemperature of the subject to generate a temperature signal thatrepresents at least one of the skin temperature or the core bodytemperature of the subject; measuring, by an oxygen saturation sensorcommunicatively coupled to the processor, blood oxygenation of thesubject to generate an oxygen saturation signal of the oxygen saturationlevel of the subject; measuring, by a photoplethysmograph (PPG) sensorcommunicatively coupled to the processor, blood flow of the subject togenerate a blood flow data signal that represents blood flow of thesubject; measuring, by an electrocardiograph (ECG) sensorcommunicatively coupled to the processor, electrical activity of a heartof the subject to generate an ECG data signal that represents electricalactivity of the heart; or measuring, by an electrodermal activity (EDA)sensor communicatively coupled to the processor, EDA of the subject togenerate an EDA data signal that represents EDA of the subject.
 14. Theelectronic device of claim 12, wherein the detecting further includes:extracting, by the processor, a plurality of audio features from thesecond audio data signal; and sub-classifying each audio feature of theplurality of audio features as indicative of a corresponding type ofsound-producing behavior.
 15. The electronic device of claim 14, whereinthe corresponding types of sound-producing behaviors include at leastone of sneezing, wheezing, shortness of breath, chewing, swallowing,masturbation, sex, coughing, vomiting, and shouting.
 16. The electronicdevice of claim 12, wherein the operations further comprise: receiving,from the subject, annotation input that confirms occurrence of one ormore sound-producing behaviors; reporting, to a remote server, theannotation input, wherein the remote server is configured to receiveannotation inputs and information about sound-producing behaviors from aplurality of subjects and to update an algorithm used in the detecting;and receiving, from the remote server, the updated algorithm.
 17. Theelectronic device of claim 12, wherein the detecting is based on theinformation derived from both the second audio data signal and thesecond data signal, the operations further comprising: deriving theinformation from both the second audio data signal and the second datasignal by converting the second audio data signal from a time domaininto a frequency domain across a plurality of frames; and discarding oneor more frames without identified sounds of interest.
 18. The electronicdevice of claim 12, wherein the microphone is oriented against a chestof the subject.
 19. The electronic device of claim 12, furthercomprising a third sensor and wherein the detecting is based on a thirddata signal from the third sensor monitoring an additionalcharacteristic other than sound.
 20. The electronic device of claim 12,wherein the operations further comprise generating a reference signalfor a plurality of sound-producing behaviors during an enrollmentprocess.
 21. The electronic device of claim 12, wherein the detectingincludes identifying a plurality of candidate sound-producing behaviors.